Electronic device, control method, and recording medium

ABSTRACT

An electronic device includes a camera, a location sensor, a display, and a controller. The controller acquires a database including name information of tangible objects, first location information indicating locations of the tangible objects, and first feature point information related to feature points of the tangible objects. The controller generates image recognition information on the basis of a captured image captured by the camera, the image recognition information including distance information indicating a distance from the electronic device to a particular subject and second feature point information related to feature points of the particular subject. The controller identifies a name of the particular subject on the basis of the image recognition information, the database, and second location information indicating a location of the electronic device acquired by the location sensor. The controller causes an overlay image, yielded by overlaying the name on the captured image, to be displayed on the display.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and benefit of Japanese PatentApplication No. 2018-032149 filed on Feb. 26, 2018, the entire contentsof which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to an electronic device, a controlmethod, and a recording medium.

BACKGROUND

A known electronic device displays an image along with date informationrelated to the image.

SUMMARY

An electronic device according to an aspect of the present disclosureincludes a camera, a location sensor, a display, a communicationinterface, and a controller. The communication interface acquires, froman external source, a database including name information of tangibleobjects, first location information indicating locations of the tangibleobjects, and first feature point information related to feature pointsof the tangible objects. The controller generates image recognitioninformation on the basis of a captured image captured by the camera, theimage recognition information including distance information indicatinga distance from the electronic device to a particular subject and secondfeature point information related to feature points of the particularsubject. The controller identifies a name of the particular subject onthe basis of the image recognition information, the database, and secondlocation information indicating a location of the electronic deviceacquired by the location sensor. The controller causes an overlay image,yielded by overlaying the name on the captured image, to be displayed onthe display.

An electronic device according to an aspect of the present disclosureincludes a camera, a location sensor, a display, a storage, and acontroller. A database including name information of tangible objects,first location information indicating locations of the tangible objects,and first feature point information related to feature points of thetangible objects is stored in the storage. The controller generatesimage recognition information on the basis of a captured image capturedby the camera, the image recognition information including distanceinformation indicating a distance from the electronic device to aparticular subject and second feature point information related tofeature points of the particular subject. The controller identifies aname of the particular subject on the basis of the image recognitioninformation, the database, and second location information indicating alocation of the electronic device acquired by the location sensor. Thecontroller causes an overlay image, yielded by overlaying the name onthe captured image, to be displayed on the display.

A method according to an aspect of the present disclosure is a methodfor controlling an electronic device comprising a camera, a locationsensor, a display, a communication interface, and a controller. Themethod includes causing, using the controller, the communicationinterface to acquire a database from an external source, the databaseincluding name information of tangible objects, first locationinformation indicating locations of the tangible objects, and firstfeature point information related to feature points of the tangibleobjects. The method includes generating, using the controller, imagerecognition information on the basis of a captured image captured by thecamera, the image recognition information including distance informationindicating a distance from the electronic device to a particular subjectand second feature point information related to feature points of theparticular subject. The method includes identifying, using thecontroller, a name of the particular subject on the basis of the imagerecognition information, the database, and second location informationindicating a location of the electronic device acquired by the locationsensor. The method includes causing, using the controller, an overlayimage yielded by overlaying the name on the captured image to bedisplayed on the display.

A non-transitory computer-readable recording medium according to anaspect of the present disclosure includes computer program instructionsto be executed by an electronic device including a camera, a locationsensor, a display, a communication interface, and a controller. Theinstructions cause the controller to cause the communication interfaceto acquire, from an external source, a database including nameinformation of tangible objects, first location information indicatinglocations of the tangible objects, and first feature point informationrelated to feature points of the tangible objects. The instructionscause the controller to generate image recognition information on thebasis of a captured image captured by the camera, the image recognitioninformation including distance information indicating a distance fromthe electronic device to a particular subject and second feature pointinformation related to feature points of the particular subject. Theinstructions cause the controller to identify a name of the particularsubject on the basis of the image recognition information, the database,and second location information indicating a location of the electronicdevice acquired by the location sensor. The instructions cause thecontroller to cause an overlay image, yielded by overlaying the name onthe captured image, to be displayed on the display.

BRIEF DESCRIPTION OF THE DRAWINGS

In the accompanying drawings:

FIG. 1 is a functional block diagram illustrating the schematicconfiguration of an electronic device according to an embodiment;

FIG. 2 is an external view of the electronic device in FIG. 1;

FIG. 3 is an external view of the electronic device in FIG. 1 from adifferent direction;

FIG. 4 illustrates an example of mounting the electronic device in FIG.1 on a bicycle;

FIG. 5 illustrates an example of a user using the electronic device inFIG. 1 while walking;

FIG. 6 illustrates an example of a settings screen;

FIG. 7 illustrates an example of the correspondence between exercise andsensor information that is overlaid;

FIG. 8 illustrates an example of another settings screen;

FIG. 9 illustrates an example of an overlay image displayed on a displayunit;

FIG. 10 illustrates another example of an overlay image displayed on thedisplay unit;

FIG. 11 illustrates yet another example of an overlay image displayed onthe display unit;

FIG. 12 illustrates an example of tangible objects included in apredetermined range;

FIG. 13 illustrates an example configuration of a database;

FIG. 14 is a flowchart illustrating processing for controlling theelectronic device (control method);

FIG. 15 illustrates an example of an overlay image displayed on thedisplay unit while settings are being made;

FIG. 16 illustrates yet another example of an overlay image displayed onthe display unit;

FIG. 17 is a flowchart illustrating processing for controlling theelectronic device (control method);

FIG. 18 illustrates yet another example of an overlay image displayed onthe display unit;

FIG. 19 illustrates yet another example of an overlay image displayed onthe display unit;

FIG. 20 is a flowchart illustrating processing for controlling theelectronic device (control method); and

FIG. 21 is a flowchart illustrating processing after an overlay image isdisplayed.

DETAILED DESCRIPTION

Techniques for displaying images and information detected by variousfunctional blocks in an electronic device have room for improvement. Itwould therefore be helpful to provide an electronic device, a controlmethod, and a recording medium that improve the convenience ofinformation display. An electronic device, a control method, and arecording medium according to embodiments of the present disclosure canimprove the convenience of information display.

FIG. 1 is a functional block diagram illustrating the schematicconfiguration of an electronic device 1 according to the presentembodiment. As illustrated in FIG. 1, the electronic device 1 includes atouchscreen display 3, a communication interface 4, an imaging unit 5, acontroller 6, sensors 7, and a storage 8. The components constitutingthese functional units are stored in or fixed to a housing 2, describedbelow.

The touchscreen display 3 is an apparatus allowing data input by thetouching of displayed images. The touchscreen display 3 includes adisplay unit 3A and an input interface 3B. The display unit 3A isprovided with a display device such as a liquid crystal display, anorganic Electro-Luminescence (EL) panel, or an inorganic EL panel. Thedisplay unit 3A displays information such as letters, images, symbols,and graphics. The information necessary for display on the display unit3A is transmitted by the controller 6. The display unit 3A may include abacklight or the like as necessary.

The input interface 3B is a touch sensor that detects contact by afinger of the user operating the touchscreen display 3, a stylus, or thelike. The input interface 3B can detect the location on the inputinterface 3B contacted by a finger, stylus, or the like. Any detectionsystem may be used in the input interface 3B, such as a capacitivesystem, a resistive film system, a surface acoustic wave system (or anultrasonic wave system), an infrared system, an electromagneticinduction system, a load detection system, or the like. The signaldetected by the input interface 3B is output to the controller 6.

The communication interface 4 transmits and receives information over acommunication network. The communication method that the communicationinterface 4 uses to transmit and receive information is prescribed by awireless communication standard. Examples of wireless communicationstandards include 2^(nd) Generation (2G), 3^(rd) Generation (3G), and4^(th) Generation (4G). Other examples of wireless communicationstandards include Long Term Evolution (LTE), W-CDMA, CDMA2000, PDC,Global System for Mobile communications (GSM® (GSM is a registeredtrademark in Japan, other countries, or both)), and Personal Handy-phoneSystem (PHS). Further examples of wireless communication standardsinclude the Standard for Wireless Fidelity (WiFi), WorldwideInteroperability for Microwave Access (WiMAX), IEEE802.11, Bluetooth®(Bluetooth is a registered trademark in Japan, other countries, orboth), Infrared Data Association (IrDA), and Near Field radioCommunication (NFC). The communication interface 4 may support one ormore of the aforementioned communication standards.

The communication interface 4 can acquire wind speed informationindicating the wind speed at the location of the electronic device 1from an external server or the like and output the wind speedinformation to the controller 6. The communication interface 4 may alsoacquire weather information, map information, mountain weatherinformation, sea weather/wave information, snow information,sunrise/sunset information, tide information, or the like and outputthis information to the controller 6. The controller 6 may store thisinformation in the storage 8 as sensor information 8C in order to handlethis information in the same way as other information output by thesensors 7.

The communication interface 4 can acquire the database 11, describedbelow, and output the database 11 to the controller 6 for storage in thestorage 8. As described below, the controller 6 can acquire the name ofa subject on the basis of the database 11 stored in the storage 8. Thecontroller 6 may store the acquired name in the storage 8 as sensorinformation 8C in order to handle the name in the same way as otherinformation output by the sensors 7. The name is used during overlayprocessing, as described below.

The imaging unit 5 is a camera for capturing images. The imaging unit 5includes at least an optical system and an image sensor. The imagesensor captures an image by conversion into an image signal of an imageof a subject formed via the optical system on an optical detectionsurface. A Charge-Coupled Device (CCD) image sensor, a ComplementaryMetal Oxide Semiconductor (CMOS) image sensor, or the like is used asthe image sensor. The image captured by the imaging unit 5 is output tothe controller 6. As described below, the controller 6 can acquire sizeinformation on the basis of the captured image. The controller 6 maystore the acquired size information in the storage 8 as sensorinformation 8C in order to handle the size information in the same wayas other information output by the sensors 7. The size information isused during overlay processing, as described below.

The sensors 7 detect information of a detection target. The sensors 7can detect changes in the detection target by continuously detecting thedetection target. The sensors 7 output the detected values to thecontroller 6. At least a portion of the detected values may be sensorinformation 8C, described below. In other words, at least a portion ofthe sensors 7 output information of the detection target to thecontroller 6 as sensor information 8C.

The sensors 7 include a location sensor 7B, a direction sensor 7C, anacceleration sensor 7D, a barometric pressure sensor 7E, and atemperature sensor 7F. A portion of these sensors may be omitted fromthe sensors 7. For example, the sensors 7 need not include thetemperature sensor 7F. The sensors 7 may also include a differentsensor.

The location sensor 7B acquires location information indicating thelocation of the electronic device 1. For example, the location sensor 7Breceives a GPS signal from a GPS satellite indicating the location ofthe GPS satellite, the transmission time of the signal, and the like.The location sensor 7B then acquires location information indicating thelocation of the electronic device 1 on the basis of the GPS signal. Thelocation sensor 7B acquires the location information continuously toacquire travel path information on the basis of a history of locationsindicated by the location information. The location sensor 7B calculatesthe change in location per unit time as the speed of the electronicdevice 1 to acquire speed information indicating the calculated speed.The location sensor 7B acquires distance information indicating thetravel distance of the electronic device 1 calculated on the basis ofthe change in location. The location sensor 7B outputs the acquiredlocation information, travel path information, speed information, anddistance information to the controller 6 as sensor information 8C.

The direction sensor 7C is, for example, a magnetic sensor or a gyrosensor. The direction sensor 7C acquires direction informationindicating the direction on the basis of a change in geomagnetism. Thedirection sensor 7C outputs the acquired direction information to thecontroller 6.

The acceleration sensor 7D acquires gravitational accelerationinformation indicating the acceleration of the electronic device 1 inthe direction of gravity. The acceleration sensor 7D outputs thegravitational acceleration information to the controller 6 as sensorinformation 8C.

The barometric pressure sensor 7E acquires barometric pressureinformation indicating the barometric pressure of the surroundingenvironment of the electronic device 1. The barometric pressure sensor7E outputs the barometric pressure information to the controller 6 assensor information 8C.

The temperature sensor 7F acquires temperature information indicatingthe temperature of the surrounding environment of the electronic device1. The temperature sensor 7F also outputs the temperature information tothe controller 6 as sensor information 8C.

The storage 8 stores a program 8A, setting information 8B, sensorinformation 8C, and image information 8D. The storage 8 may also be usedas a working area to store results of processing by the controller 6temporarily. The storage 8 may include any appropriate storage device,such as a semiconductor storage device or a magnetic storage device. Thestorage 8 may also include a plurality of types of storage devices. Thestorage 8 may include a combination of a portable storage medium, suchas a memory card, and an apparatus for reading the storage medium.

The program 8A is a program for controlling the electronic device 1. Theelectronic device 1 executes various functions by the controller 6following the instructions of the program 8A. The program 8A may beinstalled on the storage 8 through wireless communication via thecommunication interface 4 or from a storage medium. The program 8Aincludes an overlay application for the controller 6 to execute overlayprocessing. Here, overlay processing includes processing to generate anoverlay image by overlaying the sensor information 8C on an imagecaptured by the imaging unit 5. The overlay processing also includesprocessing to store the overlay image in the storage 8 as the imageinformation 8D on the basis of the setting information 8B. In thepresent disclosure, an “application” is short for an applicationprogram.

The setting information 8B is information on settings related to overlayprocessing designated by the user. The setting information 8B includesthe table described below, for example.

The sensor information 8C is information acquired by the controller 6from the sensors 7 and information generated on the basis of informationacquired by the controller 6 from the sensors 7. The sensor information8C includes information that can change over time in accordance withchanges in the state (for example, the surrounding environment) of theelectronic device 1. The sensor information 8C is, for example, locationinformation, travel path information, speed information, distanceinformation, direction information, gravitational accelerationinformation, barometric pressure information, height information,temperature information, size information, names, and the like. Thesensor information 8C is not limited to the above types of informationand may, for example, further include at least one of wind speedinformation, weather information, map information, mountain weatherinformation, sea weather/wave information, snow information,sunrise/sunset information, tide information, and the like.

The image information 8D is information of images on which overlayprocessing has been performed. The image information 8D includes stillimages and moving images. The controller 6 can, for example, read theimage information 8D from the storage 8 (such as a video on whichoverlay processing has been performed) and cause the image information8D to be displayed on the touchscreen display 3.

The controller 6 may, for example, be a Central Processing Unit (CPU).The controller 6 may be a System-on-a-Chip (SoC) or other type ofintegrated circuit in which other components, such as the communicationinterface 4, are integrated. The controller 6 may be configured bycombining a plurality of integrated circuits. The controller 6 controlsthe functional units of the electronic device 1 and implements a varietyof functions by comprehensively controlling operations of the electronicdevice 1. Details of the operations of the controller 6 in the presentembodiment are provided below.

The electronic device 1 of the present embodiment generates an image(overlay image) by overlaying sensor information 8C on a captured imageand displays the overlay image on a display as a preview image. Theelectronic device 1 of the present embodiment can store the overlayimage displayed on the display unit 3A in the storage 8. The electronicdevice 1 of the present embodiment overlays the sensor information 8C onthe captured image when the image is captured. Furthermore, theelectronic device 1 of the present embodiment stores the image with thesensor information 8C overlaid thereon. Consequently, the electronicdevice 1 of the present embodiment allows the user to reliably storecaptured images in real time after visually confirming that the sensorinformation 8C is overlaid on the captured image in a desired form(location, size, and the like).

FIG. 2 is an external view of the electronic device 1. The electronicdevice 1 may, for example, be a smartphone but is not limited to being asmartphone. The electronic device 1 may, for example, be a featurephone, a tablet, or the like. The electronic device 1 may also be animaging apparatus equipped with sensors, such as a digital camera or anaction camera.

FIG. 2 illustrates the front of the electronic device 1, where thetouchscreen display 3 is located. The housing 2 forms the outside of theelectronic device 1 and protects the components of the electronic device1, such as the touchscreen display 3, that are stored in or fixed to thehousing 2. As illustrated in FIG. 2, a portion of the imaging unit 5(for example, a front camera) is provided on the front of the electronicdevice 1. The imaging unit 5 need not be provided on the front of theelectronic device 1.

FIG. 3 illustrates the back of the electronic device 1, which is theopposite side from the front. As illustrated in FIG. 3, a portion of theimaging unit 5 (for example, a back camera) is provided on the back ofthe electronic device 1.

The electronic device 1 may, for example, be used while attached to thehandlebars of a bicycle 100, as illustrated in FIG. 4. In this case, theelectronic device 1 is attached with the imaging unit 5 facing anydirection desired by the user, for example to capture images in thetravel direction of the bicycle 100. The electronic device 1 is attachedso as to allow the user to visually confirm the touchscreen display 3.Therefore, the user can, in real time, confirm images captured by theimaging unit 5 while the user is riding the bicycle 100.

The electronic device 1 may be held and used by the user whileexercising. For example, the user can use the electronic device 1 whilewalking by holding the electronic device 1, as illustrated in FIG. 5.The user can also use the electronic device 1 by holding an auxiliaryapparatus to which the electronic device 1 is attached. The electronicdevice 1 is held so that the imaging unit 5 captures images in adirection desired by the user. The user can, for example, visuallyconfirm the touchscreen display 3 of the electronic device 1 whileholding the electronic device 1 so that the imaging unit 5 capturesimages in the user's travel direction, as illustrated in FIG. 5. Thisallows the user to visually confirm, in real time, images captured bythe imaging unit 5 while the user is exercising.

As described above, the controller 6 performs overlay processing withthe overlay application. An overlay image yielded by overlaying thesensor information 8C on the image captured by the imaging unit 5 isdisplayed on the display unit 3A by the overlay processing. The overlayimage is also stored in the storage 8 by the overlay processing. Thesensor information 8C overlaid on the captured image can be selected bythe user using the settings screen, described below.

FIG. 6 illustrates an example of a settings screen. Before thecontroller 6 starts the overlay processing, the user selects the sensorinformation 8C to be overlaid on the captured image using the settingsscreen in FIG. 6. The settings screen in FIG. 6 can, for example, bedisplayed by selection from a menu on an initial screen (top screen) ofthe electronic device 1.

The settings screen in FIG. 6 is for selecting the situation where theelectronic device 1 is to be used. Specifically, the settings screen isfor selecting what type of exercise the user will be doing when usingthe electronic device 1. The types of exercise selectable on thesettings screen in FIG. 6 are “surfing/marine sports”, “hiking”,“skiing/snowboarding”, “cycling”, “fishing”, “trekking”, and “walking”.The user selects one type of exercise by touching a radio button. Theselected type of exercise is stored in the storage 8 as one piece ofsetting information 8B.

FIG. 7 illustrates an example of the correspondence between exercise(use) and the sensor information 8C that is overlaid on the capturedimage. In the table in FIG. 7, circles indicate correspondence betweentypes of exercise and pieces of sensor information 8C. The table in FIG.7 is stored in the storage 8 as one piece of setting information 8B.

The controller 6 reads the type of exercise selected by the user fromthe setting information 8B. The controller 6 then reads the table inFIG. 7 from the setting information 8B and determines the sensorinformation 8C corresponding to the selected type of exercise. Forexample, when the type of exercise selected by the user issurfing/marine sports, the controller 6 determines that the sensorinformation 8C to be overlaid on a captured image is speed information,distance information, gravitational acceleration information, wind speedinformation, and temperature information. In other words, the controller6 displays an overlay image yielded by overlaying the speed information,distance information, gravitational acceleration information, wind speedinformation, and temperature information on a captured image. Wheninstructed by the user, the controller 6 stores the overlay image.

On the basis of the table associating exercise and sensor information 8Cto be overlaid on a captured image, the sensor information 8C isautomatically determined in accordance with the type of exerciseselected by the user. Therefore, the user can easily determine thesensor information 8C by simply selecting the situation in which theelectronic device 1 is to be used (in the present embodiment, theexercise the user is going to perform).

Here, the settings screen may use checkboxes instead of radio buttons.In this case, the user can select one or more types of exercise. Thecontroller 6 then overlays all of the sensor information 8Ccorresponding to the one or more selected types of exercise on acaptured image.

The sensor information 8C to be overlaid on the captured image need notbe restricted to the items of the sensor information 8C in theaforementioned table (speed information, distance information, heightinformation, gravitational acceleration information, barometric pressureinformation, travel path information, wind speed information, directioninformation, temperature information, size information, and names). Thecontroller 6 may further overlay current time information on thecaptured image regardless of the type of exercise selected by the user.The controller 6 may further overlay location information on thecaptured image regardless of the type of exercise selected by the user.The controller 6 may further overlay elapsed time information on thecaptured image regardless of the type of exercise selected by the user.This information indicates the time elapsed from the start of storage ofan overlay image that is a moving image.

FIG. 8 illustrates an example of another settings screen. In thisexample, the settings screen allows the user to individually select thesensor information 8C. As illustrated in FIG. 8, each type of sensorinformation 8C is displayed along with a checkbox on the settingsscreen. The user selects one or more types of sensor information 8C tooverlay on the captured image by touching the checkboxes. The selectedtypes of sensor information 8C are stored in the storage 8 as one pieceof setting information 8B. The controller 6 reads the sensor information8C selected by the user from the setting information 8B. The controller6 displays an overlay image on which the selected sensor information 8Cis overlaid. When instructed by the user, the controller 6 stores theoverlay image.

When the settings screen in FIG. 8 is used, the user can individuallyset the sensor information 8C to be overlaid on the captured image.Users can therefore customize the overlay image to suit theirpreferences.

As yet another example, the settings screen in FIG. 6 (first settingsscreen) and the settings screen in FIG. 8 (second settings screen) maybe used together. The user can cause the first settings screen and thesecond settings screen to be displayed in order using a menu on theinitial screen of the electronic device 1, for example. The user firstselects a type of exercise on the first settings screen. The selectedtype of exercise is stored in the storage 8 as one piece of settinginformation 8B. The controller 6 reads the type of exercise selected bythe user from the setting information 8B. The controller 6 then readsthe table in FIG. 7 from the setting information 8B and determines thesensor information 8C corresponding to the selected type of exercise.The controller 6 sets the checkboxes of the second settings screen to adefault selection of the sensor information 8C corresponding to theselected type of exercise. In other words, the controller 6 storesinitial values of the checkboxes of the second settings screen based onthe selected type of exercise as one piece of setting information 8B.The user then selects or deselects sensor information 8C by touching thecheckboxes on the second settings screen. The user can therefore adjustthe sensor information 8C on the second settings screen.

For example, the user may select surfing/marine sports on the firstsettings screen. The controller 6 reads the table illustrated in FIG. 7and sets the speed information, distance information, gravitationalacceleration information, wind speed information, and temperatureinformation to be initially selected (by default) on the second settingsscreen. The controller 6 then displays the second settings screen inresponse to user operation. At this time, the checkboxes for the speedinformation, distance information, gravitational accelerationinformation, wind speed information, and temperature information areselected on the second settings screen. The user can select additionalsensor information 8C or deselect sensor information 8C on the secondsettings screen. For example, if the user deselects the gravitationalacceleration information, wind speed information, and temperatureinformation, then only the speed information and distance informationremain selected. In this way, the user can use the second settingsscreen to adjust the sensor information 8C corresponding to the type ofexercise selected on the first settings screen. The user can thereforeeasily customize the sensor information 8C to be overlaid on thecaptured image.

After the sensor information 8C to be overlaid on the captured image isset, the controller 6 executes the overlay processing when the userperforms a predetermined operation to initiate the overlay processing.An example of the predetermined operation is the touching of apredetermined object displayed on the touchscreen display 3.

When instructed to initiate the overlay processing, the controller 6acquires the sensor information 8C to be overlaid on a captured imagefrom the storage 8. The controller 6 then generates an overlay image byoverlaying the sensor information 8C on the captured image. Thecontroller 6 displays the generated overlay image on the display unit3A.

FIG. 9 illustrates an example of an overlay image displayed on thedisplay unit 3A. In the example in FIG. 9, the user is riding thebicycle 100. The electronic device 1 is used while attached to thehandlebars of the bicycle 100.

The controller 6 displays an overlay image yielded by overlaying variousinformation (speed information D1, distance information D2,gravitational acceleration information D4, current time information D5,temperature information D11, and height information D12) on a capturedimage. As illustrated in FIG. 9, the various information overlaid on thecaptured image may be sensor information 8C that includes numericalinformation. The various information overlaid on the captured image maybe sensor information 8C that includes text information. The textinformation indicates the specific content of the numerical information.

For example, the gravitational acceleration information D4 illustratedin FIG. 9 is one type of sensor information 8C. The gravitationalacceleration information D4 includes the numerical information “1.2(G)”. The gravitational acceleration information D4 also includes thetext information “G-FORCE”. The numerical information “1.2 (G)” in thegravitational acceleration information D4 changes over time. The textinformation “G-FORCE” indicates that “1.2 (G)” is the value ofacceleration in the direction of gravity. The temperature informationD11 illustrated in FIG. 9, for example, is also one type of sensorinformation 8C. The temperature information D11 includes the numericalinformation 31° C. The temperature information D11 includes the textinformation “temperature”. The text information “temperature” indicatesthat “31° C.” is the temperature.

The controller 6 may arrange the information (speed information D1,distance information D2, gravitational acceleration information D4,current time information D5, temperature information D11, and heightinformation D12) at appropriate locations to avoid disturbing the user'sexercise. In the example in FIG. 9, the user is riding the bicycle 100.Information is therefore preferably not overlaid on the central andupper central portions of the display unit 3A, where the road in thedirection of travel is displayed. Rather, information is preferablydisplayed at the lower portion or near the sides of the captured image,as illustrated in FIG. 9.

Objects for user operation are also displayed in the overlay image. Thecontroller 6 causes a still image storage object O1, for example, to bedisplayed on the touchscreen display 3. The overlay image is stored as astill image when the user touches the still image storage object O1. Thecontroller 6 also causes a moving image storage object O2, for example,to be displayed on the touchscreen display 3. The overlay image isstored as a moving image when the user touches the moving image storageobject O2. The controller 6 causes an end object O3, for example, to bedisplayed on the touchscreen display 3 when storage of the moving imagebegins. Storage of the moving image ends when the user touches the endobject O3. The end object O3 may be displayed instead of the still imagestorage object O1 and the moving image storage object O2.

The controller 6 can display (reproduce) an overlay image stored in thestorage 8 as image information 8D on the touchscreen display 3. Unlikethe information overlaid on the captured image (such as the speedinformation D1), however, the objects for user operation are not storedin the storage 8. In other words, information such as the speedinformation D1 and the distance information D2, for example, isdisplayed during reproduction. Objects, on the other hand, are notdisplayed during reproduction. The overlay images stored in the storage8 as image information 8D may be acquired from an external device. Inother words, the controller 6 may display (reproduce) an overlay imageacquired from an external device through the communication interface 4on the touchscreen display 3.

FIG. 10 illustrates another example of an overlay image displayed on thedisplay unit 3A. FIG. 10 illustrates an overlay image displayed aftertime has elapsed from the state in FIG. 9. As illustrated in FIG. 10,the information overlaid on the captured image may change (in thisexample, increase) over time. The controller 6 may overlay elapsed timeinformation D6, indicating the time elapsed from the start of storage ofthe overlay image (moving image), on the captured image. The controller6 may also overlay maximum speed information D7, indicating the maximumspeed attained from the start of storage of the overlay image, on thecaptured image. The controller 6 may also overlay maximum gravitationalacceleration information D8, indicating the maximum acceleration in thedirection of gravity attained from the start of storage of the overlayimage, on the captured image. The controller 6 may also overlay travelpath information D9 on the captured image. The travel path informationD9 includes an object showing a history, based on location information,of locations in actual space (the travel path). The object is reduced tobe displayable in a predetermined region R.

FIG. 11 illustrates another example of an overlay image displayed on thedisplay unit 3A. In the example in FIG. 11, the user is walking. FIG. 11illustrates an overlay image yielded by overlaying information on animage of the surroundings captured by the imaging unit 5 while the useris walking. The overlay image in FIG. 11 includes travel pathinformation D9 displayed in a predetermined region R, temperatureinformation D11, and names D14. As described above, the travel pathinformation D9 indicates the travel path of the user in actual space asa reduced object. The temperature information D11 indicates thetemperature of the surrounding environment of the electronic device 1measured by the temperature sensor 7F. The names D14 indicate the namesof particular subjects included in the captured image. In the example inFIG. 11, the names D14 are building A and building B. As describedabove, objects for user operation such as the still image storage objectO1 and the moving image storage object O2 are displayed on the overlayimage.

Among the many buildings included in the captured image in FIG. 11, theparticular subjects indicated by the names D14 are building A andbuilding B. The particular subjects are, for example, determined asfollows.

The communication interface 4 acquires the database 11 from an externalsource in response to an instruction by the controller 6. The externalsource is, for example, an external server connected to the electronicdevice 1 over a network, such as the Internet. As illustrated in FIG.13, the database 11 includes name information 110 of tangible objects,first location information 111, which is location information indicatingthe location of tangible objects, and first feature point information112, which is information related to feature points of tangible objects.A tangible object is an object that physically occupies a portion ofspace. Tangible objects include man-made objects, such as buildings.Tangible objects also include natural objects, such as fish andmountains. The tangible objects in the database 11 are associated withparticular subjects. The particular subjects therefore include man-madeobjects. The particular subjects also include natural objects. Inanother example, the database 11 may be stored in advance in the storage8 instead of being acquired from an external source through thecommunication interface 4.

The name information 110 includes names of a plurality of tangibleobjects. When the tangible objects include buildings, for example, thename information 110 can include individual names, such as building Aand building B. When the tangible objects are fish, for example, thename information 110 can include types such as yellowtail and mackerelpike. When the tangible objects are mountains, for example, the nameinformation 110 can include individual names such as Mt. X and Mt. Y.The name information 110 may be limited to including only the names oflocally well-known buildings, commonly caught types of fish, andwell-known mountains.

The first location information 111 includes information on the locationsof a plurality of tangible objects included in the name information 110.For example, when the tangible objects include buildings, the firstlocation information 111 can include individual locations (coordinates),such as the coordinates of building A and the coordinates of building B.When the tangible objects include fish, for example, the first locationinformation 111 can include the distributions (coordinate ranges) of thehabitat of yellowtail, which is an ocean area at a water depth ofapproximately 100 m; the habitat of mackerel pike, which is an area ofthe North Pacific; and the like. When the tangible objects aremountains, for example, the first location information 111 can includeindividual locations (coordinates) such as the coordinates of Mt. X andthe coordinates of Mt. Y.

The first feature point information 112 includes information related tothe feature points of a plurality of tangible objects included in thename information 110. The first feature point information 112 includes,for example, the appearance in various types of weather, the appearancedepending on the season or time (such as morning, afternoon, and night),and the appearance (shape) from various angles (directions). In thepresent embodiment, the first feature point information 112 includesimages indicating these feature points for each of the tangible objectsincluded in the name information 110.

On the basis of an image captured by the imaging unit 5, the controller6 acquires distance information indicating the distance from theelectronic device 1 to a particular subject included in the capturedimage. In the example in FIG. 11, a captured image (preview image)including numerous buildings is displayed on the touchscreen display 3.The user designates building A as a particular subject by touchingbuilding A. As another example, the controller 6 may judge that buildingA is a particular subject by using an autofocus function when the userchanges the orientation of the electronic device 1 so that building A isincluded in the central portion of the screen. In other words, the usercan set the particular subject without touching the touchscreen display3. The controller 6 acquires distance information indicating thedistance to the particular subject using a well-known focus adjustmentfunction provided in the imaging unit 5. This function may, for example,use an infrared laser. For example, the imaging unit 5 emits an infraredlaser beam and uses a light-receiving element to receive the infraredlaser beam reflected by the particular subject. The controller 6 canthen acquire information such as the time until the imaging unit 5receives the reflected infrared laser beam to calculate the distanceinformation. The controller 6 can also acquire distance informationcalculated by the imaging unit 5.

FIG. 12 is a top view illustrating an overview of the relationship of alocation Pa of building A and a location Pb of building B in the examplein FIG. 11 to a location P0 of the electronic device 1. The controller 6acquires location information of the electronic device 1 from thelocation sensor 7B as second location information. The second locationinformation is, for example, the coordinates at location P0 of theelectronic device 1. The controller 6 identifies tangible objectsincluded in a first range R0 by identifying the first locationinformation 111 that is included in the first range R0. The first rangeR0 includes points included between the location P0 indicated by thesecond location information and locations away from the location P0indicated by the second location information by the distance indicatedby the distance information. Here, the distance indicated by thedistance information from the location P0 indicated by the secondlocation information corresponds to the straight line distance from thelocation Pa of building A in the example in FIG. 11 to the location P0of the electronic device 1. The first range R0 is a fan-shaped portionextracted, at the angle of view of the captured image, from a circlewhose center is location P0 and whose radius is the straight linedistance from location Pa to location P0. The controller 6 may useinformation on the direction indicated by the direction sensor 7C. Theshooting direction and the direction indicated by the direction sensorhave a high probability of substantially matching. In other words, thecontroller may calculate the first range R0, defined by the angle ofview of the captured image, using information on the direction indicatedby the direction sensor (which substantially matches the shootingdirection) and information on the angle of view of the captured image.The controller 6 may set the first range R0 regardless of the angle ofview of the captured image. For example, the controller 6 may set thefirst range R0 to a range with a predetermined angle width (such as 15°in each direction) to the left and right of a vector corresponding tothe direction indicated by the direction sensor. The controller 6identifies locations included in the first range R0 among the firstlocation information 111 of the database 11. In the example in FIG. 12,location Pb of building B is included in the first range R0 as well aslocation Pa. Location Pc of a building farther from location P0 of theelectronic device 1 and location Pd of a building not included withinthe angle of view of the captured image, however, are not included inthe first range R0. The controller 6 judges that location Pa andlocation Pb are included in the first range R0 and identifies thecorresponding tangible objects (building A and building B) from the nameinformation 110. In other words, building A and building B areparticular subjects in the example in FIG. 11.

On the basis of the image captured by the imaging unit 5, the controller6 then generates image recognition information including second featurepoint information related to feature points of particular subjects. Thecontroller 6 extracts the feature points of the particular subjects inthe captured image, acquires the necessary sensor information 8C, andgenerates the second feature point information. The second feature pointinformation is generated to be comparable to the first feature pointinformation 112. For example, the second feature point information isinformation that allows identification of the appearance during theweather, season, or time of shooting and the appearance (shape) in theshooting direction detected by the direction sensor 7C. The controller 6may, for example, extract the outlines of particular subjects in thecaptured image (in the example in FIG. 11, buildings A and B). Thecontroller 6 then generates the second feature point information bycombining the extracted outlines with weather information at the time ofshooting, the orientation of the captured image, or the like.

The controller 6 may use the second feature point information as is asthe image recognition information. The controller 6 may also generatethe image recognition information by adding additional information tothe second feature point information, as in the present embodiment. Forexample, the additional information included in the image recognitioninformation may be ratio information indicating the ratio of aparticular subject to the whole of the captured image. The ratioinformation can, for example, change the content of the overlay display.To increase viewability, the controller 6 may, for example, beconfigured not to overlay the name of a particular subject on thecaptured image when the particular subject has ratio information below apredetermined threshold (such as 5%). The additional informationincluded the image recognition information may be three-dimensional (3D)map information of the range included in the captured image. The 3D mapinformation clarifies the location, height, and relationships in thedepth direction of not only the particular subjects but also fixedobjects (such as buildings) and vehicles (such as cars) included in thecaptured image. The 3D map information can, for example, change thecontent of the overlay display. On the basis of the 3D map information,the controller 6 can judge whether the location of a particular subjectin the overlay image has moved, and if so, the controller 6 can changethe display location of the name overlaid on the captured image. Forexample, when the particular subject in the overlay image is a vehicle,and the location of the vehicle is judged to be moving, the controller 6may move the display location of the name in the overlay image withoutchanging the display location relative to the particular subject(vehicle). In other words, the controller 6 can generate the overlayimage on the basis of the 3D map information so that the display of aname follows a particular subject.

The controller 6 identifies the name of a particular subject in acaptured image on the basis of the image recognition information, thedatabase 11, and the second location information, which is locationinformation of the electronic device 1 acquired by the location sensor7B. The controller 6 then causes an overlay image, yielded by overlayingthe name on the captured image, to be displayed on the touchscreendisplay 3. The controller 6 uses the second location information toidentify the first range R0, as described above. Details of how thecontroller 6 uses the image recognition information and the database 11are provided below.

The controller 6 acquires the first feature point information 112 oftangible objects included in the first range R0 from the first featurepoint information 112 of the database 11. The controller 6 thenidentifies a tangible object indicating a particular subject byidentifying which of the acquired pieces of first feature pointinformation 112 matches the second feature point information included inthe image recognition information. The first feature point information112, acquired by the controller 6, of tangible objects included in thefirst range R0 can include not only the first feature point information112 of the buildings A and B but also the first feature pointinformation 112 of other buildings. The controller 6 identifies aparticular subject through a comparison with the second feature pointinformation. A shape comparison at a certain angle (shooting direction)is used for a comparison with the second feature point information inthe present embodiment, but this example is not limiting. The controller6 can use a well-known method to judge matching based on a shapecomparison.

The controller 6 designates a matching shape as a tangible objectindicating a particular subject and causes an overlay image, yielded byoverlaying the name of the tangible object on the captured image, to bedisplayed on the display. In the example in FIG. 11, the names D14(building A and building B) of the buildings at locations Pa and Pb,where the result of the shape comparison was a match, are displayed. Asillustrated in FIG. 11, the name D14 may be overlaid on the area near aparticular subject. The controller 6 need not overlay the name of asubject on the captured image when no shapes match. This embodimentallows the user to easily identify the name of a particular subjectdisplayed in the captured image. The above embodiments also allow theelectronic device 1 to identify the name of a subject to a high degreeof accuracy with a smaller processing load. Furthermore, the names ofparticular subjects are appropriately selected for display in the aboveembodiments, further improving the convenience when overlaying names ofsubjects on a captured image.

FIG. 14 is a flowchart illustrating an example method for controllingthe electronic device 1 of the present embodiment when displaying anoverlay image on which the name of a particular subject is overlaid.

The controller 6 stands by while the imaging unit 5 (camera) is notactivated (step S1: No).

When the imaging unit 5 (camera) is activated (step S 1: Yes), thecontroller 6 acquires the database 11 (step S2).

The controller 6 acquires the location information of the electronicdevice 1 from the location sensor 7B (step S3).

The controller 6 stands by while a touch by the user indicating aparticular subject is not detected (step S4: No).

When a touch is detected (step S4: Yes), the controller 6 executesprocessing for focus adjustment, i.e. focus processing (step S5).

As described above, the controller 6 acquires distance informationindicating the distance from the electronic device 1 to a particularsubject included in the captured image (step S6).

The controller 6 determines the search range of the database 11 (stepS7). In other words, by determining the first range R0, the controller 6limits the search target of the database 11 to tangible objects includedin the first range R0.

The controller 6 generates image recognition information includingsecond feature point information related to feature points of particularsubjects (step S8).

As described above, the controller 6 identifies the names of particularsubjects by shape comparison (step S9).

As illustrated in FIG. 11, for example, the controller 6 causes anoverlay image including names D14 to be displayed on the display (stepS10).

As described above, the sensor information 8C is, for example, locationinformation, travel path information, speed information, distanceinformation, direction information, gravitational accelerationinformation, barometric pressure information, height information,temperature information, size information, names, and the like. Theconvenience of the electronic device 1 further improves by two or moretypes of sensor information being overlaid on the captured image. In theexample described below, the electronic device 1 overlays the name of aparticular subject and size information in combination with other sensorinformation on the captured image.

FIG. 15 illustrates an example of an overlay image displayed on thedisplay unit 3A while settings are being made. In the example in FIG.15, the user is fishing. FIG. 15 illustrates an overlay image yielded byoverlaying information on an image, captured by the imaging unit 5, of afish the user caught. Size information, however, has not yet beenoverlaid in FIG. 15. The overlay image in FIG. 15 includes temperatureinformation D11, wind speed information D13, barometric pressureinformation D16, and location information D17. The location informationD17 indicates the current location of the electronic device 1, i.e. thelocation where the user is fishing. The wind speed information D13indicates the current wind speed, acquired by the communicationinterface 4, at the location indicated by the location information D17.The barometric pressure information D16 indicates the current barometricpressure, detected by the barometric pressure sensor 7E, at the locationindicated by the location information D17.

Objects for user operation are also displayed in the overlay image ofFIG. 15. The still image storage object O1, the moving image storageobject O2, and a focus object O4 are displayed in the example in FIG.15. As described above, the still image storage object O1 and the movingimage storage object O2 are objects for causing images to be stored inthe storage 8. In other words, the overlay image is stored as a stillimage in the storage 8 when the user touches the still image storageobject O1 (an example of a second operation). The overlay image isstored as a moving image in the storage 8 when the user touches themoving image storage object O2 (an example of a third operation). Asdescribed above, the end object O3 is displayed instead of the movingimage storage object O2, and storage of the moving image ends when theend object O3 is touched (an example of a fourth operation).

The focus object O4 is an object for starting focus processing. In otherwords, when the user touches the focus object O4 and subsequentlytouches a location, the controller 6 focuses on the object at thelocation (a fish in the example in FIG. 15) as a particular subject. Forexample, the user touches a point R1 that is a portion of a particularsubject in the captured image. The controller 6 acquires distanceinformation indicating the distance to the particular subject by using awell-known focus adjustment function provided in the imaging unit 5.This function may, for example, use an infrared laser. The user thendesignates the ends of a portion whose size the user wishes to measure.In the example in FIG. 15, the user touches point R2 (the head of thefish in the example in FIG. 15) and point R3 (the tail of the fish inthe example in FIG. 15) of the particular subject in the captured image.The controller 6 starts size information detection processing by using amethod similar to triangulation, for example, to calculate the actuallength between points R2 and R3. For example, the controller 6 canacquire information of the angle of view of images captured by theimaging unit 5. The controller 6 can also acquire the coordinates on thetouchscreen display 3 of points R2 and R3 from the input interface 3B.The controller 6 can use these pieces of information to calculate theangle between i) the shooting location and one end of the particularsubject (corresponding to point R2, for example) and ii) the shootinglocation and the other end (corresponding to point R3, for example). Thecontroller 6 can then calculate the actual length using the distanceinformation to the particular subject acquired during focus adjustment.The length calculated in this way is size information.

FIG. 16 illustrates an example of an overlay image displayed on thedisplay unit 3A after FIG. 15. As described above, the controller 6overlays the size information on the captured image when the userperforms an operation designating the ends of a portion whose size theuser wishes to measure (an example of an operation to prescribe ameasurement range within a captured image that includes a certainimaging area). The controller 6 displays, on the display unit 3A, anoverlay image yielded by overlaying an object indicating the sizeinformation on the captured image that includes the particular subject.In the example in FIG. 16, the object is a gauge D15. The gauge D15includes an image of a line segment having a predetermined width. Thegauge D15 includes numbers and units (cm) to indicate the length. Thecontroller 6 may further display size information on the overlay imagein a different form than the object. In the example in FIG. 16, theoverlay image includes a measurement result display D18, which is sizeinformation.

On the basis of an image captured by the imaging unit 5, the controller6 thus measures size information related to a particular subjectincluded in the captured image and displays, on the display unit 3A, anoverlay image yielded by overlaying both the size information (such asthe gauge D15) and sensor information 8C based on the detection resultof the sensors 7 (such as the location information D17) on the capturedimage. On the basis of an image captured by the imaging unit 5, thecontroller 6 also measures size information related to a particularsubject included in the captured image and displays, on the display unit3A, an overlay image yielded by overlaying both the size information(such as the gauge D15) and information acquired by the communicationinterface 4 (such as the temperature information D11, wind speedinformation D13, and barometric pressure information D16) on thecaptured image. As described above, the information acquired by thecommunication interface 4 may be treated as sensor information 8C. Inthe example in FIG. 16, the controller 6 measures size informationrelated to the whole of the particular subject and displays, on thedisplay unit 3A, an overlay image yielded by overlaying an objectindicating the size information on the captured image that includes thewhole of the particular subject. The controller 6 can measure the sizeinformation of at least a portion of the particular subject. The usermay designate a portion instead of the whole of the particular subjectwhen designating the ends of the portion whose size the user wishes tomeasure.

The controller 6 may cause the overlay image to be displayed on thedisplay unit 3A on the basis of operation of the imaging unit 5. Forexample, when the user performs an operation to activate the camera(such as touching a camera icon on the main menu, which is an example ofthe first operation), the controller 6 may display the overlay image onthe display unit 3A. As described above, after displaying the overlayimage on the display unit 3A, the controller 6 stores the overlay imagein the storage 8 as a still image when the user touches the still imagestorage object O1.

The controller 6 may overlay the sensor information 8C (such as thelocation information D17) on the captured image for only a predeterminedtime when the user performs an operation to activate the camera. Thecontroller 6 may overlay an image related to size information (such asan image of the line segment having a predetermined width in the gaugeD15) on the captured image for only a predetermined time when the userperforms an operation to activate the camera. Display of the imagerelated to the size information allows the user to recognize that sizeinformation can be measured.

As described above, after the user performs an operation to designatethe ends of the portion whose size the user wishes to measure, thecontroller 6 may store the overlay image with the size informationoverlaid thereon in the storage 8 as a still image when the user touchesthe still image storage object O1. In other words, the controller 6 neednot overlay the size information on the captured image when the userperforms the operation to designate the ends of the portion whose sizethe user wishes to measure. Rather, the controller 6 may overlay thesize information on the captured image when storing the overlay image inthe storage 8 as a still image in response to the user touching thestill image storage object O1.

The particular subjects in the above embodiment may include a pluralityof subjects of the same type with individual differences in size. Theparticular subject may be a man-made object or a natural object. Theparticular subject may be a living organism or an inanimate object.Examples of particular subjects that are inanimate objects includebuildings and furniture. Examples of particular subjects that are livingorganisms are not limited to fish but also include plants and animals.

The above embodiment allows the user to easily identify the size of aparticular subject displayed in the captured image. The above embodimentalso allows the user to learn what sized organisms can be caught orfound in a certain environment. Furthermore, the above embodiments allowthe user to confirm or recall what sized organisms were caught or foundin a certain environment after capturing images. The size of a subjectmay have a causal relationship with the surrounding environment. Forexample, it might be easy to catch large fish at a particular locationamong a large, unspecified number of locations. Furthermore, thetemperature, wind speed, and barometric pressure change depending on thetime of day and season, but it may be easy to catch large fish at aparticular temperature, wind speed, or barometric pressure. The usercan, for example, record and learn what sized organism is likely toinhabit a certain environment. When the user captures an image of a fishhe caught, it may therefore be useful to display the size, name, or thelike of the caught fish on the captured image. The above embodimentsalso improve operability of the electronic device 1 by overlaying sizeinformation related to the particular subject.

FIG. 17 is a flowchart illustrating an example method for controllingthe electronic device 1 of the present embodiment when displaying anoverlay image on which the size information of a particular subject isoverlaid.

The controller 6 stands by while the imaging unit 5 (camera) is notactivated (step S101: No).

When the imaging unit 5 (camera) is activated (step S101: Yes), thecontroller 6 proceeds to step S102.

The controller 6 stands by while a touch by the user indicating aparticular subject is not detected (step S102: No).

When a touch is detected (step S102: Yes), the controller 6 executesprocessing for focus adjustment, i.e. focus processing (step S103). Inthe above example, the controller 6 detects a touch by acquiringinformation indicating that the user touched the focus object O4 fromthe input interface 3B.

As described above, the controller 6 measures the size information of atleast a portion of the particular subject (step S104).

As in FIG. 16, for example, the controller 6 causes an overlay imageincluding the gauge D15 and the measurement result display D18 to bedisplayed on the display (step S105).

The controller 6 can include the name of the particular subject and thesize information in the overlay image. FIG. 18 illustrates an example ofan overlay image displayed on the display unit 3A. In the example inFIG. 18, the user is walking. As compared to FIG. 11, the overlay imagein FIG. 18 further includes the gauge D15 and the measurement resultdisplay D18. With the same method as above, the user designates thebottom and top of building A and causes the controller 6 to measure thelength (i.e. the height of building A). In the example in FIG. 18, theheight of building A is 55 m. On the basis of an image captured by thecamera, the controller 6 thus measures size information related to atleast a portion of a particular subject (building A) included in thecaptured image and displays an overlay image yielded by overlaying boththe size information and name information on the captured image. Thename D14 is specified on the basis of the tangible object nameinformation 110 acquired by the communication interface 4. In otherwords, on the basis of an image captured by the camera, the controller 6measures size information related to at least a portion of a particularsubject (building A) included in the captured image and displays anoverlay image yielded by overlaying both the size information andinformation acquired by the communication interface 4 (name D14) on thecaptured image.

FIG. 19 illustrates an example of an overlay image displayed on thedisplay unit 3A. In the example in FIG. 19, the user is fishing. Ascompared to FIG. 16, the overlay image in FIG. 19 further includes aname D14. In the example in FIG. 19, the name D14 is “yellowtail”, thetype of fish that the user caught. The controller 6 includes, in theoverlay image, at least one of information shared by the particularsubject (name D14) and information of the individual size of theparticular subject (gauge D15 and measurement result display D18). Thecontroller 6 identifies the information shared by the particular subjectin the following way, for example. First, the controller 6 acquires thelocation information of the electronic device 1 from the location sensor7B. The controller 6 judges that the particular subject is a fish usingwell-known image processing or the like. The controller 6 selects acandidate type of fish depending on whether the ocean near the locationinformation of the electronic device 1 is included in the habitatindicated as the first location information 111. The controller 6 thenidentifies the name by a shape comparison between the first featurepoint information 112 of the candidate fish and the fish included in thecaptured image. As illustrated in FIG. 19, both name information of theparticular subject included in the captured image and sensor information(information acquired by the communication interface 4) may be overlaidon the captured image.

In the above embodiments, both name information and size information, orboth name information and sensor information (information acquired bythe communication interface 4) are overlaid on the captured image andstored. After storage, the user can therefore newly confirm or recallnot only the name but also other information necessary for the useralong with the captured image. For example, the user can collectivelyrecord required information, such as images of structures, names, anddates, at a construction site or the like and confirm the informationlater.

FIG. 20 is a flowchart illustrating an example method for controllingthe electronic device 1 of the present embodiment when displaying anoverlay image on which the name and size information of a particularsubject are overlaid.

The controller 6 stands by while the imaging unit 5 (camera) is notactivated (step S201: No).

When the imaging unit 5 (camera) is activated (step S201: Yes), thecontroller 6 acquires the database 11 (step S202).

The controller 6 acquires the location information of the electronicdevice 1 from the location sensor 7B (step S203).

The controller 6 stands by while a touch by the user indicating aparticular subject is not detected (step S204: No).

When a touch is detected (step S204: Yes), the controller 6 executesprocessing for focus adjustment, i.e. focus processing (step S205).

As described above, the controller 6 acquires distance informationindicating the distance from the electronic device 1 to a particularsubject included in the captured image (step S206).

The controller 6 determines the search range of the database 11 (stepS207). In other words, the controller 6 limits the search target of thedatabase 11 on the basis of the location information of the electronicdevice 1 or the like.

The controller 6 generates image recognition information includingsecond feature point information related to feature points of theparticular subject (step S208).

As described above, the controller 6 identifies the name of theparticular subject by shape comparison (step S209).

As described above, the controller 6 measures the size information of atleast a portion of the particular subject (step S210).

As in FIG. 18 or FIG. 19, for example, the controller 6 causes anoverlay image including the names D14, gauge D15, and/or measurementresult display D18 to be displayed (step S211).

In this way, the electronic device 1 according to the present embodimentcan cause overlay images such as those described above to be displayedon the display, improving convenience of information display.

Next, an example of processing after display of the overlay image isdescribed with reference to the flowchart in FIG. 21.

The electronic device 1 displays the overlay image on the display (stepS1001).

When a touch on the still image storage object O1 is detected (stepS1002: Yes), the electronic device 1 stores an overlay still image (stepS1003).

When a touch on the still image storage object O1 is not detected (stepS1002: No), the electronic device 1 proceeds to step S1005.

When a touch on the moving image storage object O2 is detected (stepS1005: Yes), the electronic device 1 starts to store an overlay movingimage (step S1006).

When a touch on the moving image storage object O2 is not detected (stepS1005: No), the electronic device 1 proceeds to step S1004.

From the start of storage of the overlay moving image, the electronicdevice 1 stands by while recording the overlay moving image as long as atouch on the end object O3 is not detected (step S1007: No).

When a touch on the end object O3 is detected (step S1007: Yes), theelectronic device 1 ends storage of the overlay moving image (S1008) andproceeds to step S1004.

The electronic device 1 ends the processing sequence when a touch on themoving image storage object O2 is not detected after storage of theoverlay still image, or when an end operation is detected after storageof the overlay image is complete (step S1004: Yes).

When an end operation is not detected (step S1004: No), the electronicdevice 1 returns to step S1001.

Although the subject matter of the present disclosure has been describedwith reference to the drawings and embodiments, it is to be noted thatvarious changes and modifications will be apparent to those skilled inthe art on the basis of the present disclosure. Therefore, such changesand modifications are to be understood as included within the scope ofthe present disclosure. For example, the functions and the like includedin the various components, steps, and the like may be reordered in anylogically consistent way. Furthermore, components, steps, and the likemay be combined into one or divided.

For example, the user can make various settings regarding the display ofthe name D14 on the overlay image. The user may, for example, specifythat the name D14 not be overlaid on a designated tangible object in thecaptured image. When the user designates a particular building, forexample, a portion of the name D14 “building B” that was displayed inthe foreground may be removed from the foreground, as in FIG. 18.

The name D14 may be overlaid only on tangible objects designated by theuser. In other words, when a predetermined operation including anoperation to select one or more particular subjects is performed afterthe controller 6 causes the captured image to be displayed on thedisplay unit 3A, the controller 6 may cause an overlay image, yielded byoverlaying the name D14 of each selected particular subject on thecaptured image, to be displayed on the display unit 3A. At this time,the user can control which names D14 are displayed on the overlay imageand can thus prevent the overlay image from becoming difficult to seebecause of too many names D14. The controller 6 may provide notificationwhen no match occurs during shape comparison. For example, when no matchoccurs during shape comparison, the controller 6 causes an overlayimage, yielded by overlaying text or an object indicating the lack ofmatching on the captured image, to be displayed on the display. Theobject indicating the lack of matching during shape comparison may behidden after being overlaid on the captured image for a predeterminedtime. The text indicating the lack of matching during shape comparisonmay, for example, be “unknown name”. The object indicating the lack ofmatching during shape comparison may, for example, be a question mark.This configuration allows the user to identify whether a name was notoverlaid on the captured image because overlay processing was notexecuted or because overlay processing was executed but was unable toidentify the name of the subject.

As another example, the name D14 may be overlaid with the followingmethod only on tangible objects designated by the user. When the namesof a plurality of particular subjects included in the captured image areidentified, as in the example in FIG. 11, the controller 6 causes afirst overlay image, yielded by overlaying the names of the plurality ofparticular subjects on the captured image, to be displayed on thedisplay. When the user performs a predetermined operation including anoperation to select one or more particular subjects after the firstoverlay image is displayed, the controller 6 causes a second overlayimage, yielded by hiding the name of the selected one or more particularsubjects, to be displayed on the display. In this example, informationon each particular subject selected by the predetermined operation isdetected by the input interface 3B and output to the controller 6. Thecontroller 6 then hides the name of each selected particular subject.

After the user touches the focus object O4 and the point R1 in theexample in FIG. 15, the focus object O4 may be removed from the overlayimage, and the user may be notified that the mode has changed. In otherwords, when a predetermined operation (such as touching the focus objectO4) is performed while the captured image is being displayed, thecontroller 6 may switch between a first mode in which focus processingcan be performed and a second mode in which size information detectionprocessing can begin.

In the example in FIG. 12, tangible objects whose location is includedin the first location information 111 (buildings A and B) are present inthe first range R0. As described above, however, the probability of suchtangible objects not being present in the first range increases when thecontroller 6 sets the first range R0 with no relation to the angle ofview of the captured image as compared to when the controller 6 sets thefirst range R0 in accordance with the angle of view of the capturedimage. When no such tangible object is present in the first range R0,the controller 6 may perform the following processing. When no tangibleobject is present in the first range R0, the controller 6 identifiesfirst location information 111 included in a second range that includespoints included between the location indicated by the second locationinformation (the location P0 of the electronic device 1) and locationsaway from the location indicated by the second location information (thelocation P0 of the electronic device 1) by the distance indicated by thedistance information, without regard to direction. In other words, thecontroller 6 draws a circle centered on the location P0 of theelectronic device 1 and identifies tangible objects included in theregion enclosed by the circle (second range). A portion of the secondrange corresponds to the first range R0. The controller 6 identifies atangible object indicating a particular subject by identifying which ofthe pieces of first feature point information 112 of the tangibleobjects included in the second range matches the second feature pointinformation. The controller 6 then causes an overlay image, yielded byoverlaying the name representing the tangible object indicating theparticular subject on the captured image, to be displayed on thedisplay. The controller 6 may execute this processing not only when notangible object whatsoever is present in the first range R0, but alsowhen no first feature point information 112 matches the second featurepoint information during the shape comparison processing (when at leastone name is unknown during the shape comparison processing).

When requesting that the user perform a new operation, the controller 6in the above embodiments may include a guide in the overlay image toprompt the user. For example, after the user touches the focus object O4in the example in FIG. 15, the controller 6 may display “Tap a locationto focus on” to prompt the user to tap a particular subject. After theuser then touches the fish, which is a particular subject, in theexample in FIG. 15, the controller 6 may display “Tap the ends of thefish” to encourage the user to designate the length to measure. Theconvenience of the electronic device 1 is further improved by such adisplay prompting the user for the next operation.

In another embodiment, the controller 6 may generate the overlay imageand then store the overlay image in the storage 8 without displaying theoverlay image on the display unit 3A. In other words, the controller 6may generate the overlay image and then store the overlay image in thestorage 8 while displaying a captured image on which overlay processinghas not been performed on the display unit 3A.

The functions provided in the electronic device 1 may be implemented bya plurality of apparatuses. For example, the functions provided in theelectronic device 1 may be implemented by a first apparatus thatincludes at least the imaging unit 5 and a second apparatus thatincludes at least the display unit 3A. The first apparatus and thesecond apparatus are connected in a wired or wireless manner. The firstapparatus transmits at least an image captured by the imaging unit 5 tothe second apparatus in a wired or wireless manner. Either the firstapparatus or the second apparatus may perform the overlay processing onthe captured image. When the first apparatus performs the overlayprocessing, the first apparatus transmits the overlay image to thesecond apparatus in a wired or wireless manner. The second apparatusdisplays the received overlay image on the display unit 3A. When thesecond apparatus performs the overlay processing, the first apparatustransmits a captured image, on which overlay processing has not beenperformed, and the sensor information 8C to the second apparatus in awired or wireless manner. The second apparatus performs the overlayprocessing on the received captured image and displays the overlay imageon the display unit 3A. The first and second apparatuses may, forexample, be an action camera and a smartphone, or an action camera and awearable device, but these examples are not limiting.

The invention claimed is:
 1. An electronic device comprising: a camera;a location sensor; a display; and a controller configured to acquire adatabase including name information of tangible objects, first locationinformation indicating locations of the tangible objects, and firstfeature point information related to feature points of the tangibleobjects, generate image recognition information on the basis of acaptured image captured by the camera, the image recognition informationincluding distance information indicating a distance from the electronicdevice to a particular subject and second feature point informationrelated to feature points of the particular subject, identify a name ofthe particular subject on the basis of the image recognitioninformation, the database, and second location information indicating alocation of the electronic device acquired by the location sensor, andcause an overlay image, yielded by overlaying the name on the capturedimage, to be displayed on the display.
 2. The electronic device of claim1, further comprising a communication interface configured to acquirethe database from an external source, wherein the controller isconfigured to acquire the database via the communication interface. 3.The electronic device of claim 1, further comprising a storage in whichthe database is stored, wherein the controller is configured to acquirethe database stored in the storage.
 4. The electronic device of claim 3,wherein the controller is configured to store the overlay image in thestorage as a still image when a second operation is performed after thecontroller causes the overlay image to be displayed on the display. 5.The electronic device of claim 3, wherein the controller is configuredto start processing to store the overlay image in the storage as amoving image when a third operation is performed after the controllercauses the overlay image to be displayed on the display, and end theprocessing to store the overlay image as the moving image when a fourthoperation is performed after the controller starts the processing tostore the overlay image as the moving image.
 6. The electronic device ofclaim 1, wherein the particular subject includes a natural object. 7.The electronic device of claim 1, wherein the controller is configuredto identify the tangible objects included in a predetermined range andidentify a tangible object indicating the particular subject from amongthe tangible objects included in the predetermined range, and cause anoverlay image, yielded by overlaying the name representing the tangibleobject indicating the particular subject on the captured image, to bedisplayed on the display.
 8. The electronic device of claim 7, whereinthe controller is configured to identify the tangible objects includedin a first range by identifying the first location information includedin the first range, the first range including points included betweenthe location indicated by the second location information and locationsaway from the location indicated by the second location information bythe distance indicated by the distance information, identify thetangible object indicating the particular subject by identifying thefirst feature point information matching the second feature pointinformation from among the first feature point information of thetangible objects included in the first range, and cause an overlayimage, yielded by overlaying the name representing the tangible objectindicating the particular subject on the captured image, to be displayedon the display.
 9. The electronic device of claim 8, further comprisinga direction sensor, wherein the controller is configured to identify thetangible objects included in the first range by identifying the firstlocation information included in the first range, the first rangeincluding points included between the location indicated by the secondlocation information and locations away from the location indicated bythe second location information by the distance indicated by thedistance information in a direction indicated by the direction sensor,identify the tangible object indicating the particular subject byidentifying the first feature point information matching the secondfeature point information from among the first feature point informationof the tangible objects included in the first range, and cause anoverlay image, yielded by overlaying the name representing the tangibleobject indicating the particular subject on the captured image, to bedisplayed on the display.
 10. The electronic device of claim 9, whereinthe controller is configured to identify the tangible objects includedin a second range by identifying the first location information includedin the second range when no tangible object is included in the firstrange, the second range including points included between the locationindicated by the second location information and locations away from thelocation indicated by the second location information by the distanceindicated by the distance information without regard to the direction,identify the tangible object indicating the particular subject byidentifying the first feature point information matching the secondfeature point information from among the first feature point informationof the tangible objects included in the second range, and cause anoverlay image, yielded by overlaying the name representing the tangibleobject indicating the particular subject on the captured image, to bedisplayed on the display.
 11. The electronic device of claim 9, whereinthe controller is configured to identify the tangible objects includedin a second range by identifying the first location information includedin the second range when the first feature point information of thetangible objects included in the first range does not include firstfeature point information matching the second feature point information,the second range including points included between the locationindicated by the second location information and locations away from thelocation indicated by the second location information by the distanceindicated by the distance information without regard to the direction,identify the tangible object indicating the particular subject byidentifying the first feature point information matching the secondfeature point information from among the first feature point informationof the tangible objects included in the second range, and cause anoverlay image, yielded by overlaying the name representing the tangibleobject indicating the particular subject on the captured image, to bedisplayed on the display.
 12. The electronic device of claim 1, whereinthe controller is configured to cause the overlay image to be displayedon the display when a first operation is performed.
 13. The electronicdevice of claim 1, wherein the overlay image comprises a first overlayimage and a second overlay image, and the controller is configured tocause the first overlay image to be displayed on the display when namesof a plurality of particular subjects included in the captured image areidentified, the first overlay image being yielded by overlaying thenames of the plurality of particular subjects on the captured image, andcause the second overlay image to be displayed on the display when apredetermined operation including an operation to select one or more ofthe particular subjects is performed after the first overlay image isdisplayed, the second overlay image being yielded by hiding the name ofthe selected one or more of the particular subjects.
 14. The electronicdevice of claim 1, wherein the controller is configured to cause thecaptured image to be displayed on the display, and cause the overlayimage to be displayed on the display when a predetermined operationincluding an operation to select one or more of particular subjects isperformed after the captured image is displayed on the display, theoverlay image being yielded by overlaying the name of the selected oneor more of the particular subjects on the captured image.
 15. Theelectronic device of claim 1, wherein the image recognition informationincludes ratio information indicating a ratio of the particular subjectto whole of the captured image, and the controller does not overlay thename of the particular subject on the captured image when the particularsubject has ratio information below a predetermined threshold.
 16. Theelectronic device of claim 1, wherein the image recognition informationincludes three-dimensional map information of a range included in thecaptured image.
 17. The electronic device of claim 16, wherein thecontroller is configured to judge, on the basis of the three-dimensionalmap information, whether location of the particular subject in theoverlay image has moved, and change a display location of the nameoverlaid on the captured image when it is judged that the location ofthe particular subject has moved.
 18. The electronic device of claim 17,wherein the controller is configured to move the display location of thename in the overlay image without changing a relative display locationof the name relative to the particular subject when the location of theparticular subject in the overlay image has moved.
 19. The electronicdevice of claim 18, wherein the name is overlaid near the particularsubject.
 20. The electronic device of claim 19, wherein the name is notoverlaid on the particular subject.
 21. A method for controlling anelectronic device comprising a camera, a location sensor, a display, acommunication interface, and a controller, the method comprising:causing, using the controller, the communication interface to acquire adatabase from an external source, the database including nameinformation of tangible objects, first location information indicatinglocations of the tangible objects, and first feature point informationrelated to feature points of the tangible objects; generating, using thecontroller, image recognition information on the basis of a capturedimage captured by the camera, the image recognition informationincluding distance information indicating a distance from the electronicdevice to a particular subject and second feature point informationrelated to feature points of the particular subject; identifying, usingthe controller, a name of the particular subject on the basis of theimage recognition information, the database, and second locationinformation indicating a location of the electronic device acquired bythe location sensor; and causing, using the controller, an overlay imageyielded by overlaying the name on the captured image to be displayed onthe display.
 22. A non-transitory computer-readable recording mediumincluding computer program instructions to be executed by an electronicdevice comprising a camera, a location sensor, a display, acommunication interface, and a controller, the instructions causing thecontroller to: cause the communication interface to acquire a databasefrom an external source, the database including name information oftangible objects, first location information indicating locations of thetangible objects, and first feature point information related to featurepoints of the tangible objects; generate image recognition informationon the basis of a captured image captured by the camera, the imagerecognition information including distance information indicating adistance from the electronic device to a particular subject and secondfeature point information related to feature points of the particularsubject; identify a name of the particular subject on the basis of theimage recognition information, the database, and second locationinformation indicating a location of the electronic device acquired bythe location sensor; and cause an overlay image, yielded by overlayingthe name on the captured image, to be displayed on the display.